DOC  FILE  COPY  ADA07  6634 


DEGREE  OF  TRAINING  AND  ARTILLERY 
CONTROL  CONSOLE  OPERATOR 
(ACCO)  PROFICIENCY 


Douglas  Griffith 


ARI  FIELD  UNIT  AT  FORT  HOOD,  TEXAS 


D  D  C 


1  DISTRIBUTION  STATEMENT  X  | 

Approved  for  public  release;  . 

OintTihution  Unlimited  1 

iiPfTorpnnniPJ 

~n\ 

11  MO V  16  1979 

u 

UlialMtjlJ  Li  Lbl 

A 


U  S.  Army 

Research  Institute  lor  the  Behavioral  and  Social  Sciences 


February  1979 

7  9  11  15  5 


U.  S.  ARMY  RESEARCH  INSTITUTE 

FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 

A  Field  Operating  Agency  under  the  Jurisdiction  of  the 
Deputy  Chief  of  Staff  for  Personnel 

WILLIAM  L.  HAUSER 

JOSEPH  ZEIDNER  Colonel,  US  Army 

Technical  Director  Commander 


NOTICES 


DISTRIBUTION  Primary  distribution  ot  this  report  has  bean  made  by  ARI.  Please  address  correspondence 
concerning  distribution  ot  reports  to:  U.  S.  Army  Research  Institute  tor  the  Behavioral  and  Social  Sciences, 
ATTN  PERI-P,  5001  Eisenhov^r  Avenue,  Alexandria.  Virginia  22333 


Thu  report  may  be  destroyed  when  it  is  no  longer  needed.  Please  do  not  return  it  to 


the  U.  S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 


NOTE  The  findings  in  this  report  are  not  to  be  construed  as  an  official  Department  of  the  Army  position, 
unless  so  designated  by  other  authorized  documents. 


Army  Project  Number 
|  [^\2Q263743A775 


Human  Performance  in 
Field  Assessment 


.4) 


Research  Problem  Review  79-3 


J^GREE  OF  RAINING  AND  ARTILLERY  CONTROL  CONSOLE 
OPERATOR  (ACCO)  PROFICIENCY  ^  ^ 


JJyj  Dpuglas^Griffith 


Submitted  by: 

George  M.  Gividen,  Chief 
ARI  FIELD  UNIT  AT  FORT  HOOD,  TEXAS 


r 


Feb*tary.’**79 


{  If) 

ARI-RBS  PROBLBM  RKV-79-3 

! 

I 


Approved  by: 


A.  H.  Birnbaum,  Acting  Director 
Organizations  and  Systems 
Research  Laboratory 


Joseph  Zeidner,  Technical  Director 
U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences 


Research  Problem  Reviews  are  special  reports  to  military  management. 
They  are  usually  prepared  to  meet  requirements  for  research  results  bear¬ 
ing  on  specific  management  problems.  A  limited  distribution  is  made— 
primarily  to  the  operating  agencies  directly  involved. 


O'XO  Jy 


FOREWORD 


i 

The  Fort  Hood  Field  Unit  of  the  Army  Research  Institute  (ARI)  for 
the  Behavioral  and  Social  Sciences  conducts  research  and  provides  tech¬ 
nical  advisory  service  for  the  Training  and  Doctrine  Command  (TRADOC) 
Combined  Arms  Test  Activity  (TCATA) .  In  the  summer  of  1977,  the  Com¬ 
manding  General  of  TRADOC  requested  that  ARI  conduct  a  performance 
evaluation  of  TACFIRE  Artillery  Control  Console  Operators  (ACCOs)  to 
serve  as  an  input  to  the  cost  training  effectiveness  analysis  (CTEA) 
of  the  TACFIRE  system.  TACFIRE  is  a  newly  developed  automated  command 
and  control  system  for  the  field  artillery.  ACCOs  function  at  the 
computer  consoles  located  at  the  battalion  and  division  artillery  Fire 
Detection  Centers  and  are  integral  to  the  overall  effectiveness  of  the 
TACFIRE  system. 


The  entire  project  is  responsive  to  special  requirements  of  the 
office  of  the  Deputy  Chief  of  Staff  for  Personnel  and  to  Army  Project 
2Q263743A775 . 
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BRIEF 


Requirement : 


To  assess  the  proficiency  of  TACFIRE  ACCOs  as  a  function  of  amount 
of  training  with  the  TACFIRE  equipment.  \ 


Procedure : 

Operators'  proficiency  at  the  artillery  control  console  (ACC)  was 
assessed  at  their  graduation  from  formal  TACFIRE  training  and  after 
2*5  months  and  5*5  months  of  on-the-job  training  (OJT)  .  A  practical  ex¬ 
ercise  consisting  of  10  processing  requirements  for  the  ACCO  (e.g., 
inputting  targets  into  the  battalion  target  file,  processing  a  search 
of  the  division  artillery  intelligence  file,  establishing  a  fire  plan) 
was  developed.  Eight  of  the  processing  requirements  were  timed  for 
each  operator,  and  all  processing  requirements  were  divided  into  sub¬ 
requirements,  which  were  scored  for  accuracy.  Operators  were  run  in¬ 
dividually  in  the  TACFIRE  computer  shelters. 


Findings : 

No  differences  were  found  among  the  three  groups  with  respect  to 
processing  accuracy.  However,  enlisted  operators  were  significantly 
faster  after  having  received  OJT  than  at  graduation.  No  significant 
differences  were  obtained  between  the  2*5  and  5*s  month  OJT  groups. 


Utilization  of  Findings: 


These  findings  impact  on  future  TACFIRE  ACCO  training.  Findings 
have  been  integrated  into  TRADOC's  cost  training  effectiveness  analy¬ 
sis  (CTEA)  of  the  TACFIRE  system. 
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INTRODUCTION 

TACFIRE  is  a  computerized  command  and  control  system  for  the 
field  artillery.  At  the  heart  of  the  TACFIRE  system  is  the  artillery 
control  console  (ACC) .  ACCs  are  located  at  the  division  artillery 
and  the  battalion  fire  direction  centers  (FDCs) .  Although  the  fire 
direction  sergeant  has  primary  responsibility  for  operating  the  ACC, 
other  FDC  personnel  (e.g.,  fire  direction  officer,  TACFIRE  equipment 
specialist)  typically  take  turns  at  the  console  during  tactical  oper¬ 
ations.  Because  the  FDC  is  critical  to  TACFIRE  system  performance, 
an  extensive  training  program  was  administered  to  all  FDC  personnel 
prior  to  TACFIRE  OT  056.  FDC  personnel  received  10  weeks  of  formal 
course  training  at  Fort  Sill,  Okla.,  prior  to  entering  a  dedicated, 
on-the-job  (OJT)  training  phase  at  Fort  Hood,  Tex.  This  OJT  phase 
lasted  until  the  onset  of  operational  testing.  Approximately  half 
the  OT  056  FDC  personnel  completed  their  formal  training  at  the  be¬ 
ginning  of  June  1977.  The  remaining  FDC  personnel  completed  their 
formal  training  at  the  beginning  of  September  1977.  Hence,  a  differ¬ 
ence  of  approximately  3  months  total  OJT  time  existed  between  the  two 
groups . 

A  question  of  interest  to  both  test  personnel  and  to  individuals 
performing  the  cost  training  effectiveness  analysis  (CTEA)  of  the 
TACFIRE  system  for  TRADOC  was  whether  or  not  there  was  a  detectable 
performance  difference  that  was  attributable  to  this  3-month  differ¬ 
ence  in  training  time. 


OBJECTIVE 

The  objective  of  this  research  was  to  assess  ACCO  proficiency  as 
a  function  of  the  degree  of  training  with  the  TACFIRE  equipment. 


METHOD 

A  practical  test-*-  was  developed,  and  it  could  be  administered  to 
an  individual  operator  in  half  an  hour.  The  test  consisted  of  10  re- 
t  quirements  (e.g.,  inputting  targets  into  the  battalion  target  file, 

processing  a  search  of  the  division  artillery  intelligence  file, 


The  test  is  controlled,  since  it  will  be  administered  in  future  test¬ 
ing.  Information  about  the  test  can  be  requested  by  writing  Chief, 

ARI  Field  Unit,  HQ  TCATA  (PERI-OH) ,  Fort  Hood,  Tex.  76544. 
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establishing  a  fire  plan) .  The  first  two  requirements  comprised  mes¬ 
sages  sent  to  the  operator.  For  these  first  two  requirements,  oper¬ 
ators  were  asked  questions  about  the  messages  and  asked  to  take  ap¬ 
propriate  action.  The  remaining  eight  requirements  required  specific 
operator  actions  and  were  timed  for  each  operator.  All  10  processing 
requirements  were  divided  into  subrequirements  which  were  stringently 
scored  on  a  go /no-go  basis. 

The  initial  test  was  administered  at  Fort  Hood  in  mid-November 
1977  by  the  training  assistance  team  from  Fort  Sill.  This  test  was 
conducted  simultaneously  at  the  four  battalion  FDCs  and  at  the  di¬ 
vision  artillery  FDC.  A  subsequent  iteration  of  the  test  was  admin¬ 
istered  at  Fort  Sill  by  the  TACFIRE  instructors  on  the  USAFAS  computers 
in  early  April  1978.  This  subsequent  iteration  was  run  to  establish  a 
performance  baseline  at  graduation  from  formal  TACFIRE  training. 

The  basic  rationale  underlying  the  study  was  to  compare  the  per¬ 
formance  of  these  three  classes  to  determine  whether  or  not  there  were 
differences  in  operator  proficiency  attributable  to  training  experi¬ 
ence.  The  three  points  of  evaluation  were  at  TACFIRE  graduation  (the 
Fort  Sill  test  in  April  1978) ,  after  2*s  months  of  postgraduation  OJT 
(the  September  1977  class) ,  and  after  5*3  months  of  postgraduation  OJT 
(the  June  1977  class) . 


RESULTS 

The  results  are  portrayed  in  Figures  1  and  2 .  Summaries  of  the 
statistical  analyses  are  presented  in  the  Technical  Supplement.  These 
statistical  analyses  (a  =  .05)  revealed  that  the  differences  among  the 
three  classes  with  respect  to  performance  accuracy  were  not  statisti¬ 
cally  significant.  However,  statistical  analyses  of  the  processing 
time,  when  restricted  to  enlisted  personnel,  did  reveal  statistically 
significant  differences.  Further  analyses  indicated  that  whereas  the 
processing  times  for  the  June  and  September  classes  were  significantly 
faster  than  the  processing  times  for  the  graduating  class,  the  process¬ 
ing  times  of  the  former  two  classes  did  not  differ  significantly  from 
each  other.  The  small  numbers  of  officers  represented  in  the  sample, 
coupled  with  their  disproportionate  distribution  among  the  classes, 
precluded  any  meaningful  analysis  of  officer  proficiency. 

Analyses  were  also  performed  on  TACFIRE  course  grades  to  ascer¬ 
tain  whether  the  classes  differed  in  aptitude  or  general  knowledge. 
These  analyses  revealed  no  significant  differences  in  the  school  per¬ 
formance  of  these  three  classes. 
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CONCLUSIONS 


The  primary  conclusion  is  that  by  the  time  enlisted  graduates 
of  the  Fort  Sill  Fire  Support  Course  have  completed  2h  months  of  in¬ 
tensive  hands-on  training,  operator  proficiency  has  essentially  leveled 
off.  Another  conclusion  is  that  the  skills  of  individual  operators  im¬ 
proved  during  the  initial  phase  of  OJT.  The  precise  point  at  which 
operators'  skills  level  off  remains  to  be  determined  empirically. 

A  number  of  caveats  must  be  extended  concerning  the  above  con¬ 
clusions.  Assumptions  regarding  the  reliability  and  validity  of  the 
test  are  discussed  in  the  Technical  Supplement.  Note  also  that  the 
above  conclusions  are  specific  to  enlisted  personnel  and  are  specific 
to  individual  operator  proficiency.  For  example,  the  fact  that  indi¬ 
vidual  operator  proficiency  leveled  off  does  not  necessarily  imply  a 
corresponding  leveling  off  of  overall  system  proficiency.  Only  spe¬ 
cific  individual  operator  tasks  were  assessed.  Neither  FDC  proficiency 
as  a  team  nor  proficiency  in  the  field  was  assessed.  Certain  special 
properties  of  team  performance  (e.g.,  the  quality  of  supervision)  and 
special  problems  encountered  in  the  field  (e.g.,  communications)  can 
profoundly  affect  system  performance.  Moreover,  these  factors  them¬ 
selves  can  be  modified  by  training. 


TECHNICAL  SUPPLEMENT 


Preliminary  analyses  were  conducted  to  assess  the  comparability 
of  the  three  classes.  Mean  class  averages  were  83%,  85%,  and  84%  for 
the  April  1978  (Graduation) ,  September  1977  (24  month  OJT) ,  and  June 
1977  (54  month  OJT)  classes,  respectively.  An  unweighted  means  ANOVA 
revealed  no  significant  differences  among  the  three  classes,  F(2,  41) 

<  1.00.  Because  the  April  1978  class  contained  a  disproportionately 
high  percentage  of  officers  (42%  versus  30%  and  27%  for  the  September 
1977  and  June  1977  classes,  respectively),  it  was  deemed  prudent  also 
to  conduct  the  analyses  employing  only  the  scores  of  enlisted  person¬ 
nel.  Mean  class  averages,  when  restricted  to  enlisted  personnel,  were 
79%,  82%,  and  80%  for  the  April  1978  (Graduation),  September  1977 
(2*5  month  OJT),  and  June  1977  (54  month  OJT)  classes,  respectively. 

The  unweighted  means  ANOVA  of  these  scores  again  revealed  nonsignifi¬ 
cant  effects,  F(2,  26)  <  1.00.  Hence,  it  was  concluded  that  subsequent 
analyses  could  be  legitimately  related  to  training  conditions. 

The  means  percent  incorrect  were  14.93,  12.10,  and  14.74  at  gradu¬ 
ation,  and  after  24  months  and  54  months  of  postgraduation  OJT,  re¬ 
spectively.  The  unweighted  means  ANOVA  of  these  scores  indicated  that 
the  obtained  differences  were  nonsignificant,  F(2,  41)  <  1.00.  When 
officers  were  excluded  from  the  preceding  analyses,  the  means  were 
21.36%,  13.57%,  and  17.00%  at  graduation,  and  after  24  and  54  months 
of  OJT  following  graduation,  respectively.  Again,  the  unweighted 
means  ANOVA  revealed  nonsignificant  effects,  F(2,  26)  =  1.024,  p  >  .25. 
Thus,  total  training  time  had  no  significant  effect  on  operator 
accuracy. 

The  mean  response  times  were  120  seconds,  107  seconds,  and  104 
seconds  at  graduation,  24  months,  and  54  months,  respectively.  The 
unweighted  means  ANOVA  of  these  times  revealed  a  nonsignificant  ef¬ 
fect,  F(2,  41)  =  2.37,  p  >  .10.  However,  when  officers  were  excluded 
from  this  analysis,  the  effect  became  significant,  F(2,  26)  =  3.91, 
p  <  .05.  When  restricted  to  enlisted  personnel,  the  mean  response 
times  were  131  seconds,  109  seconds,  and  107  seconds  at  graduation, 

24  months,  and  54  months,  respectively.  A  Newman-Keuls  test  (a  =  .05) 
of  these  means  indicated  that,  whereas  the  classes  at  24  months  and 
54  months  were  significantly  faster  than  the  class  tested  at  gradua¬ 
tion,  the  classes  at  24  and  54  months  were  not  significantly  differ¬ 
ent  from  each  other.  Thus,  the  speed  of  performance  of  the  enlisted 
operators  did  improve  during  OJT.  However,  after  24  months,  no  fur¬ 
ther  improvement  in  speed  was  detected. 


Winer,  B.  J.  Statistical  Principles  in  Experimental  Design.  New 
York:  McGraw-Hill,  1971. 


Additional  analyses  were  also  done  on  a  natural  logarithm  trans¬ 
formation  of  the  response  times.  In  one  set  of  analyses,  each  response 
time  for  each  requirement  was  transformed  to  its  natural  logarithm, 
and  then  the  mean  of  the  transformed  scores  was  employed  as  the  de¬ 
pendent  variable.  The  unweighted  means  ANOVA  of  these  scores  revealed 
a  significant  effect,  F(2,  41)  =  3.78,  p  <  .05.  The  means  of  the  trans¬ 
formed  scores  were  4.481,  4.234,  and  4.267  at  graduation,  2*s  months, 
and  5*j  months,  respectively.  The  Newman-Keuls  test  (a  =  .05)  of  these 
means  again  indicated  that  whereas  the  classes  at  2*5  months  and  at 
5*5  months  were  significantly  faster  than  the  class  tested  at  gradua¬ 
tion,  the  classes  at  2*5  months  and  at  5*5  months  were  not  significantly 
different  from  each  other.  The  same  pattern  of  results  obtained  when 
officers  were  excluded  from  the  analysis,  F(2,  26)  =6.28,  p  <  .01 
(x  =  4.579,  4.223,  and  4.288  at  graduation,  2*s  months,  and  5*s  months, 
respectively) . 

A  second  set  of  analyses  was  done  on  the  natural  logarithmic  trans¬ 
formation  of  the  mean  processing  times  for  each  operator.  This  analy¬ 
sis  is  not  identical  with  the  preceding  one,  because  the  mean  of  a  non¬ 
linear  transformation  is  not  the  same  as  the  nonlinear  transformation 
of  the  mean.  The  unweighted  means  ANOVA  of  the  transformation  of  the 
mean  scores  revealed  a  nonsignificant  effect,  F(2,  41)  =  2.31,  p  <  .05. 
The  means  of  the  transformed  means  were  4.776,  4.652,  and  4.625  at 
graduation,  2*5  months,  and  5*5  months,  respectively.  However,  when  of¬ 
ficers  were  excluded  from  the  analysis,  the  effect  became  significant, 

F (2 ,  26)  =  4.02,  p  <  .05.  With  the  officers  excluded,  the  means  be¬ 
came  4.869,  4.669,  and  4.658  at  graduation,  2*5  months,  and  5*s  months, 
respectively.  The  Newman-Keuls  test  (a  =  .05)  indicated  that,  whereas 
the  first  mean  was  significantly  different  from  the  others,  the  latter 
two  means  did  not  differ  significantly  from  each  other.  Note  that  the 
pattern  of  results  from  this  set  of  analyses  is  identical  with  that  of 
the  untransformed  scores. 

Given  the  above  analyses,  the  prudent  course  is  to  restrict  gen¬ 
eralizations  about  the  data  to  enlisted  personnel.  Since  officers, 
on  the  whole,  tended  to  perform  at  a  higher  level  than  enlisted  per¬ 
sonnel,  and  since  officers  were  not  represented  proportionately  in 
each  group,  generalizations  with  respect  to  officers  remain  highly 
tenuous.  Moreover,  only  one  of  the  analyses  in  which  officers  were 
included  yielded  significant  effects.  However,  the  results  regarding 
enlisted  personnel  are  quite  firm — the  speed  of  operator  performance 
did  improve  after  intensive  periods  of  OJT,  and  this  speed  leveled  off 
by  the  time  operators  had  received  2*s  months  of  intensive  OJT. 

Of  course,  generalizing  from  the  results  of  the  above  test  to 
"true"  operator  proficiency  requires  a  high  degree  of  test  validity. 

At  this  time  it  is  uncertain  how  representative  this  test  is  of  true 
operator  proficiency.  Unfortunately,  the  time  constraints  imposed 
upon  the  study  and  the  limited  number  of  TACFIRE-trained  personnel 
precluded  a  rigorous  procedure  of  test  development  inclusive  of  formal 
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assessments  of  reliability  and  validity.  Although  the  test  appears 
to  have  excellent  face  validity,  it  is  recommended  that  validity  be 
properly  assessed  in  the  future.  Moreover,  it  is  recommended  that  a 
pool  of  similar  tests  be  developed  for  the  assessment  of  operator 
proficiency. 
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